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Abstract 



Project Care Phase II was a field evaluation of a com- 
munication and learning system package of films, simulation 
games, discussion questions, and posters designed to promote 
career awareness in junior high school children. It is pro- 
posed that this evaluation serve as a prototype for the as- 
sessment of effects of learning systems on the affective and 
cognitive domains of target audiences. In the experimental 
testing design, 1460 children in selected fifth, seventh, 
and ninth grade classes in three Texas cities of varying sizes 
underwent training with the use of the system. Experimental 
and control classes were tested before and after the former 
classes were exposed to the materials. Results indicated that 
the materials had a significant effect upon increases in 
knowledge about different careers , knowledge on where to get 
training for those careers, and upon increased positive attitudes 
about job importance and job favorability . These effects had 
generality across pupil grade levels, sex, ethnicity, parental 
occupational and educational levels, and teachers’ attitudes 
about the materials. A component experiment indicated that 
results were essentially the same for ninth grades (the only 
tested) wnether or not teachers had undergone training in the 
use of the materials. Subjective questions indicated a high 
degree of favorability of the p v upils toward the materials. 
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Abstract 

Project Care Phase II was a field evaluation of a training 

package of films, simulation games, discussion questions, and 

. VM' 0 / 

posters designed to promote career awarenesses m grade school 
children. In an experimental testing design, ^children in selected 
fifth, seventh, and ninth grade classes in Houston, Port Arthur, 
and Conroe underwent training with use of the package. Experi- 
mental and control classes were tested before and after the 
former classes underwent training. Results indicated that the 
materials had a significant effect upon increases in knowledge 
about different careers, knowledge on where to get training 
for those careers, an d upon increased positive attitudes about 
job importance, and job favorability . There were no effects 
upon measures of seeing oneself in such jobs. The above effects 
had generality across pupil grade levels, sex, ethnicity, 
parental occupational and educational levels, and teachers* 
attitudes about the materials. A component experiment indicated 
that results were essentially the same for ninth grades (the 
only tested) whether or not teachers had undergone training 
in the use of the materials. Subjective questions indicated 
a h:igh degree of favor of the pupils toward the materials. 



Introduction 



Objectives 

Project Care, Phase I involved the development of The 
Occupational Awareness Multi - Media Package, , which consisted of 
films, simulation games, discussion questions, and posters. The 
basic purpose of this project, Phase II, was to determine the 
effectiveness of the package by conducting a series of field 

experiments . 

Evaluation data were gathered in fifth, seventh, and ninth 
grade classes in schools sampled in Houston, Port Arthur, and 
Conroe, Texas. Within each school, classes were divided between 
experimental groups, which underwent career awareness training 
with the package, as against control groups, who were tested but 
did not receive the training materials. Within each test area 
several additional ninth grade classes were tested where, although 
the class underwent career awareness training, their teachers 
did not receive in-service training in the use of. the materials. 

Evaluation materials included subtest measures of five 
key variables representing knowledge and attitudes associated 
with the aims of the training packages. These subtests were: 

1 # knowledge about different careers 

2 . knowledge of where to get training for these careers 

3. favorability of attitude toward the importance 
of these careers 

4 . favorability of attitude about the careers themselves 

5. favorability of attitude of seeing oneself in one 



of these careers 
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Secondary information was obtained on pupils' attitudes about 
the materials as well as teachers' evaluation which were obtained 
at the conclusion of in-service training. 

Problem 

The specific aims of the evaluation design were to answer 
the following questions: 

Effects of materials : What are the effects of the materials 

upon occupational knowledge and attitudes for each of the fifth, 
seventh, and ninth grades, and how do such effects vary across 

test cities? 

Pupil differences : How do effects of the materials vary 

according to such variables as pupil sex, ethnicity, parent 
education and occupation, and teachers' attitudes toward the 

materials? 

Pupils ' evaluations : What are pupils' evaluations of the 

package, the films, games, and posters, and what are their 
attitudes toward this type of instruction? 

Effects of teacher training : What are the effects upon 

occupational knowledge and attitudes, and upon pupils' evaluation 
of materials due to having .teachers trained in use of the materials? 

Teachers ' evaluations : What were teachers' evaluations of 

the materials and of in-service training? 

Organization of the Report 

To facilitate presentation of the overall results, this 
report is arranged in order from the general to the specific. 

The next section presents the najor results of the research. 
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This is followed by a more detailed description of the research 
method. A technical appendix contains summaries of statistical 
details as well a sample copies of testing materials. 



( 






i 



7 



7 

Overall Results 

Effects of Ma terials : What are the effeate of the materials 

” • 

upon oooupational knowledge and attitudes for each of the fifth, 
seventh, and ninth grades, and how do such \ef feats vary across 
test cities? 

Subtest gains in all cities combined . In the analyses of 
combined data, consistent occupational knowledge gains and 
increased favorability of attitudes were found in the fifth, 
seventh, and ninth grades as a function of undergoing occupational 
awareness training. In the following sub tests, the average 
gains from pre- to post -tests were significantly (see Technical 
Appendix for statistical details) greater for experimental 
groups (receiving training) than control groups (no training): 
Occupational knowledge, training knowledge, attitude of job 
importance, and attitude of job favorability. Only on the 
sub test of favorability of oneself in given jobs were there 
no appreciable gains nor differences between experimental and 
control groups. 

Table 1 summarizes these results in graphic form. Results 
are arranged so as to emphasize comparisons between control 
and experimental groups for each subtest and for each grade. 
Asterisks indicate the comparisons where differences in gains 
between experimental and control groups were statistically 
significant. 

The combined data also showed a slight tendency for some 
differences in subtest gains according to grades. Ninth grades 
tended to gain more in training knowledge than did fifth and 
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Table: 1 Score Gains : All Cities Combined. 



Score 



Fifth 

Grade 



Seventh 

Grade 



cont . exp. 



cont. exp. 



Ninth 

Grade 



cont. exp* 



‘3 



.■I 

> 



Occupational. 

knowledge^ 



(' 



Training 

knowledge. 



Attitude of 
job iapor; 
tance 





Scale 



. *3 • 
• ’♦' 2 . 
.’••♦.I- 
. 0 
. -1 
•-2 




■i ; 

3 



■i ) 



j 



Attitude of 
self in_ 
job 
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•Statistically aifaificant (p<.05) di£far*nca batwaan 
control and experimental groupgaina. 
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seventh grades. Seventh and ninth grades increased more than 
fifth grades in terms of ratings of job favorability. These 
two findings can be interpreted as modest evidence that the 
materials, at least in terms of two measures in the overall 
analyses, were slightly less effective with the younger children. 
However, on two other measures --occupational knowledge and' 
job importance --there were no differences. 

In all, there was abundant evidence in the overall 
analyses that the materials had effects on four of the five 
main assessment measures. 

Sub test gains in Houston . The Houston analyses indicated 
that experimental, groups gained significantly more than control 
groups in all three grades on the two measures of occupational 
knowledge and training knowledge. Further, gains in experimental 
groups exceeded control groups for the seventh and ninth grades 
on measures of attitude of job importance and attitude of job 
favorability. As in the overall analyses, there were no signifi? 
cant gains on the measure of attitude of self in job. 

Table 2 is a graph • summary of the Houston results. In 
the most general terms it indicates that the materials were more* 
effective in seventh and ninth grades as compared with fifth 
grades. Additionally, the materials appear to have no effects 
upon increasing favorability of pupils' attitudes about seeing 
themselves in selected careers. 

Subtest gains in Port Arthur . Again, analyses indicated 
that the materials had a significant effect upon gains in 
occupational knowledge and training knowledge across all three 
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Table: 2 Score Gains: Houston 



Fifth 

Grade 



Score 



.cont. exp. 



Occupational 

knowledge 



Seventh 

Grade 



cont. exp. 



10 



Ninth 

Grade 



cont. exp. 



• • * 



Training 
knowledge 



C.\- 



Attitude of 
job iapor; 
tance 






w 

* 


,4 

1 .* 






1 


t 

• 




■“T== 


1 — 


1 1 





Scale 



. +3 
. *2 

-♦1 
_ 0 

. -1 
. *2 
. -3 



Attitude of 
job fav - • 
orability L 
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grades and no effects upon attitude of oneself in various jobs. 

Results were mixed in terms of effects upon attitude of job 
importance and attitude of job favorability . Here significant 
gains in the experimental groups over control groups were found 
for fifth and seventh grades in terms of attitude of job 
importance, but not in ninth grade. Finally, only fifth grades 
showed an effect of the training materials in terms of attitude 
of job favorability. 

Again results are summarized in graphic form (Table 3). 

One feature unique to the Port Arthur results was that if any 
overall grade differences were to be found, they slightly 
favored the fifth and seventh grades over the ninth grades. 

Similar to the other analyses, knowledge gains were the most 
consistent effects of the training, and attitude of oneself 
.. in particular jobs was unchanged. 

Sub test gains in Conroe . Results of the testing in Conroe 
indicated significant gains by the experimental groups over the 
I ( control groups in the three measures: Occupational knowledge, 

training knowledge-, and attitude of job importance. Increases 
in attitude of job favorability were found for seventh and ninth 
grades, but not for fifth grades. Again, no effects were found 
in terms of attitude toward oneself in various jobs. 

Table 4 is a summary of the Conroe results. Here there was , 

no overall tendency for some grades to do.better than others; the 
picture was mixed. Knowledge gains, however, are clear cut, | 

* •' " ty 

and attitude effects were as great or greater than in the other two M 
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Fifth 

Grade 



Score 



cont. exp. 



Occupational 

knowledge 



Training 
knowledge. 



Attitude of 
job iapor^_ 
tance ““ 



Attitude of 
job fav- 



orabilify" 



self in 
job 






Seventh 

Grade 



Ninth 

Grade 



cont. exp. 



cont. exp. 




, \ 



•Statistically significant (p<.0S) difference between 

iins. •• 



control and experimental group ga 
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Scale 
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♦2 ...i 

* • 

♦1 
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r 1 
-2 
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Fifth 

Grade 



Seventh 

Grade 



Ninth 

Grade 



Score 



cont. exp* 



cont. exp. 



cont. exp. 



Training 
knowledge_ 



t * ' . 



Attitude of 
job import 
tance ” 



t 

. ’-I 






Attitude of 
job fav- • 
or ability 



Attitude of 
self in 
job 



. . Occupational . 
knowledge 




i-' — 










■ i_ 


> • — 


• 




i • 

* 



JZZL 




•Statistically significant (p«.0$) difference between 

gains* 



control and experimental group _ 

„*• . ••• v * ..*• •*. - '.*•*• ' * .*<£* • . V-. • 
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♦3 

*2 
♦1 
. 0 

. -1 
-2 
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Some generalizations : 

1. The training package had the most consistent effects 
upon increases of knowledge about various occupations and 
increases in knowledge of where to get training for these 
occupations. These effects have generality across the three 
test cities. 

2. Changes in attitudes toward the importance of jobs 
mentioned in the occupational awareness training package were 
generally consistent and favorable across all test cities and 
grades. Only the fifth grades in Houston and the ninth grades 
in Port Arthur failed to show changes. 

3. Changes in attitudes toward the favorability of 
selected careers were found for the most part in all cities, 
but with more exceptions than attitudes about: job importance. 

No pattern of these differences seemed evident. 

4. The materials apparently had no effects upon changes 
of attitude about oneself in various occupations. There were 
very few pre-to post -test changes and no differences between 
experimental and control groups. 

5. There was a slight tendency for seventh and ninth 
grades to show more consistent effects of occupational training. 
However, this was not a major difference, and it was some- 
what reversed in the Port Arthur results. 



Pupil Differences : How do effects of the materials vary according 

to euoh variables as pupil sex, ethnicity, parent education and 
occupation, and teachers' attitudes toward the materials? 

Gains re lative to pupi 1 sex . Both male and female pupils 
in experimental groups gained significantly more than their 
counterparts in the control groups in all subtests except 
attitudes of self in job (which had not been significant in 
any analyses) . In gains in occupational knowledge and training 
knowledge, female pupils who received training did slightly 
better on the average than did male pupils. However, there 
were no differences between male and female students in the 
effects of the materials on attitude of job importance and job 
favorability.. 

Table 5 summarizes the sex difference by comparing experi- 
mental and control groups divided in terms of sex in the over- 
all data from all cities combined. In this table it can be 
seen that the effects of the materials have considerable 
generality across pupil sex. 

Gains relative to pupi 1 ethnicity . With only one exception, 
all of the subtest gains found for the overall analyses of data 
from all cities were also found when analyses were done for 
control and experimental group comparisons within Black, Anglo, 
and Mexican-American ethnic groups. That is, on occupational 
knowledge, training knowledge, attitude of job importance, and 
attitude of job favorability, subgroups of Black, Anglo, and 
.Mexican-American students who were trained with the package 
exceeded in gains over comparable groups who did not undertake 






Table 5: Score Gains, Male as against Female 
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Score 



Male | Female 

cont. exp. cont. exp. 



Occupational 

knowledge 




( 



Training 

knowledge 



Attitude of 
job impor; 
tance 






Scale 

. 3 
, 2 

. i 
-0 
.-i 

-2 
r 3 



Attitude of 
job fav- 
orabilify" 



Attitude of 
self in 
job 
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*Statistically significant (g<.05) difference between 
control and experimental group gains. 
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the training. The one exception was that Black pupils in the 
experimental groups showed no statistically significant gain 
differences over the control groups in measures of attitude 
of job favorability . 

There was a slight but consistent and significant tendency 
for Anglo students in experimental groups to show greater gains 
than Black students. Mexican-American pupils, however did not - 
fit consistently into such a pattern; they sometimes gained 
more, or sometimes less, than their counterparts. 

Table 6 summarizes the gain scores in terms of ethnic 
groups. The general pattern of experimental groups exceeding 
control groups on the subtests can be seen in this table, as 
can the generality of findings across ethnic groups. Again, 
it can be noted that no groups gained on the measure of 
attitude of self in job. 

Gains relative to parent occupation and education . Responses 
to the question of parent occupation were quantified in order 
to establish a numerical scale. To determine whether gains 
due to the training materials might be related to parent 
occupation, a correlation index was calculated between the 
numerical scaling of parent occupations and the gain scores 
for pupils. If this index were approximately .075 or greater, 
it would be statistically significant in this case; however, 
its magnitude would have to be upward of .40 or greater to be 
of practical concern here. The calculated correlations were: 
(.'10) occupational knowledge; (.11) training knowledge; (.01) 
job importance; (.04) job favorability; and (-.06) self in job. 




Table 6: Score Gains , Ethnic Differences 
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Score 



Black Anglo Mex. Am. 

cont. exp. cont. exp. cont. exp. 



( 



( 



Occupational 

knowledge_ 




Training 

knowledge 














‘ T 




\ 





Attitude of 

job impor- i • | [___^ 

tance ! [ 



Scale 

* 3 

. 2 
. 1 
> 0 
. -1 
. -2 
. -3 



* 



* 



Attitude of 

job fav- 1 I ^ 

orability p ' T 

\ I 



Attitude of 
self in 
job 






I 



L 
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•Statistically significant (|3<.0S) difference between 
control and experimental group gains. 
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The interpretations of these results were as follows: 

There is a statistically significant, but very slight, if 
negligible relation, between level of parent occupation and 
pupils' gains in occupational knowledge and training know- 
ledge. This one might expect a very slight chance that, say, 
a son of a physician .might gain more in knowledge scores from 
the training materials than the son of a laborer. There was 
no significant relation between parent occupational level and 
pupils' gains in ratings of job importance, job favorability, 
or self in job. In other words, gains in these attitudinal 
ratings appeared independent of the occupational levels of 
the pupils' parents. 

Relations of gains with the parents' educational levels 
was examined in the same manner. Responses to questions on 
mother and father educational levels were averaged to provide 
one overall scale of parental educational level. Correlational 
indexes were calculated between this scale and the five main 
subtest gain scores. The same bases for interpreting the 
correlations as discussed above applied in this case. 

Results indicated the following correlations with the 
parental educational level: ( . 02) occupational knowledge; 

(.01) training knowledge; (-.06) job importance; (-.04) job 
favorability; (-.06) s'jlf in job. Since none of these corre- 
lation indexes was statistically significant, it was concluded 
that gains in the fiye subtest measures appeared independent 
of the educational levdl of parents. 
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Gains relative to teacher attitudes . 'At the completion 
of training in use of the career awareness' materials, teachers 
had been given a questionnaire which included items pertaining 
to their evaluation of the materials and their expectations 
of success. Prom these responses, a teacher attitude scale was 
developed. To explore the relation between pupils' gain 
scores and teacher attitudes, a correlation index was calculated 
and interpreted as discussed above . 

Correlations in this case were: (.02) occupational 

knowledge; (.Q0) training knowledge; (.00) job importance; 

(.09) job favorabi lity; (.074) self in job. Interpretations 
were: There was no apparent relation between teacher attitude 

toward the materials and gains in occupational knowledge, 
training knowledge, nor ratings of job importance. There 
were statistically significant, but very small, if not 
negligible, relations between teacher attitude and gains in 
ratings of job favorability and self in job. These meant 
that if a teacher were favorably disposed toward the materials, 
there might be a very slight effect that her pupils would gain 
more in ratings of job favorability and self in job. Again, 
however, this relation is small enough to be treated as probably 
negligible . 

Some generalizations : 

1. The effects of the training materials were similar for 
male and female pupils in measures of gains in occupational 
knowledge, training knowledge, job importance, and job favor- 
ability. However, females did slightly better in gains on the 
knowledge measures. 

_2L_ ; 
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2. For the most part, gains in occupational knowledge, 
training knowledge, job importance, and job favorability had 
generality across Black, Anglo, and Mexican-American pupils. 
Anglo students, however, had a very slight tendency to gain 
more than Black students. 

3. There were no important relations of parent occupa- 
tional level or educational level with pupils' gains from use 
of the training package. 

4. There were no important relations between teachers' 
attitudes toward the materials and the effects of the training 
package upon pupils. 
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Pupils * Evaluations : What are pupils' evaluations of the 

package , the films, gamee, and poetere , and what are their 
attitudes toward this type of instruction? 

Tabular summary . Table 7 summarizes in terms of proportional 
breakdowns the pupils' responses to questions about the training. 
These proportions are calculated for a breakdown of responses 
by grades, by sex, and by ethnicity. Each proportion repre- 
sents the relative number of responses to each multiple response 
item for a question. Thus, for example, in Table 7 there is 
a proportional breakdown of the different responses of fifth 
graders to question number 130 on how good, pupils thought the 
materials were for students their age. Here, .54 means that 54 
percent of the fifth graders responding to the question felt 
that the materials were about right for their age. 

Table 7 may be consulted directly for a detailed summary 
of results; however, the following will offer a briefer 
summary. 

Age rating . In all cases the majority of pupils thought 
that the materials were "about right" for people of their ages. 
Ratings tended to become more favorable as children were from 
higher grades, and for females as compared with males. There 
were apparently no ethnic differences. 

Rating t he t eacher . Most pupils rated their teachers as 
doing "an especially good job" in making the best use of the 
training materials. Fifth graders gave somewhat more positive 
ratings on this. There were no major sex differences. Mexican- 
Americans tended to rate their teachers somewhat higher than 
did the other two groups. 

- 2 ? 
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Use of media materials in other classes . Most students 
gave favorable responses to the question of whether they 
would "learn more" if subjects such as math, English and social 
studies used games, posters, and films. There were no major 
differences in responses to this question by grades nor sex. 

Black and Mexican-American students did tend to answer this 
question more favorably than did Anglo students. 

Best film . Among the three grades, the film on health care 
{ received the highest ratings, however it was closely followed 

by the film on criminal justice. Female pupils rated health 
care higher than criminal justice, whereas males rated criminal 
justice the higher of the two. Mexican-American pupils rated 
criminal justice higher than health care, whereas Black and 
Anglo students rated health care the higher. 

Best game . All grades, both sexes, and all three ethnic 
groups rated the criminal justice game as the best. This was 
followed by the game on aerospace. 

Best poster . The criminal justice poster was generally 
rated as the best. However, this selection was not a majority 
one; it averaged approximately one- fourth of the responses. 

Materials learned the most from . The majority of students 
listed the films as the materials that they learned the most 
from. This was generally the same across all' grade, sex and 
ethnic categories. 

Most interesting materials . Films were the most consistent 
first choice, but these were closely followed by the games. 
Findings were similar across all grade, sex and ethnic categories 
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except for Mexican -American students who rated films substantially 
higher than the other materials. 

Most boring materials . Class discussions and posters were 
the most frequently listed as the most boring materials. These 
detailed findings varied somewhat across the different categories, 
but the overall pattern was consistent across grade, sex, and 
ethnic categories. 

Comparison with other subjects . Students tended to rate 
the training materials as "more interesting" than other 
materials used in school. This tended to diminish, however, 
in comparisons from fifth to ninth grade. Females rated the 
materials higher than males. Anglo and Mexican -American 
students rated the materials higher than did Blacks. 

Learned the most about . Criminal justice and health care 
received the highest ratings of careers learned the most about. 
These results were generally consistent across grades, sexes, 
and ethnic groups. 

Learned the least about. Computer science was the topic 
most often rated as having learned the "least about." This, 
also, had consistency across grades, sexes, and ethnic groups. 

Some generalizations ; 

1. Pupils in all grades, of both sexes, and from Black, 
Anglo, and Mexican -American ethnic groups rated the instructional 
materials typically in a very favorable manner. 

2. Most pupils thought that their teachers had done a 

good job with the materials. 
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3. Pupils indicated a favor for using such materials with 
other subjects . 

4. Criminal justice and health care materials as well 
as instructional results received the highest ratings. 

5. Mass communication tended to receive the lowest 



ratings 
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Effects of Teacher Training : What are the effeote upon 

oooupational knowledge and attitudes , and upon pupils * evaluation 
of materials , due to having teachers trained in use of the 

materials? 

Effects upon score gains . Pupils 1 average score gains in 
ninth experimental groups having trained teachers were compared 
with those having untrained teachers on each of the five subtests. 
Results , as summarized in Table 8, indicated that there were no 
statistically significant differences between these mean scores. 

It may be recalled that these comparisons were made in different 
cities only between the ninth grade classes. In short, teacher 
training for ninth grades appeared to have no effect upon over- 
all score gains by experimental groups. 

Effects upon pupil e valuations . Pupils' responses to questions 
about the instructional materials were divided between the ninth 
grade groups who had trained as against untrained teachers. 

These response comparisons are summarized in Table 9. In general, 
they show very few differences between responses by pupils having 
trained as compared with untrained teachers. There was a slight 
tendency for pupils having trained teachers to rate their teacher 
as doing a "better job" in making use of the instructional 
materials, and a stronger tendency for them to rate the criminal 
justice materials as the ones they "learned the most" from. 

Again, however, these differences were slight, and probably 
negligible. 

Generalizations : 

1. • Teacher training had no apparent effect upon 
score gains in the five main subtests for ninth graders. 
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2. Only negligible differences were found in pupils' 
evaluations of materials when compared between ninth grades having 
trained as against untrained teachers. 
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Table 8 : Score Gains: Trained as against Untrained Teachers 



Score 



Trained Untrained 

Teachers Teachers 



Occupational 

knowledge^ 



Training 

knowledge 



Attitude of 
job impor^ 
tance 



Scale 

. 3 
. 2 
1 

_ 0 
. -1 

. -2 
. -3 



Attitude of 

job fav- : 

orability 



Attitude of 

self in g - - --I < L 

job r ' 




•Statistically significant (^<.05) difference between 
control and experimental group gains. 
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Table 9: Pupils' Evaluations, Trained and Untrained Teachers 



130. 



131. 



132. 



Questions Teachers 

Train'd Untrained 



I thought that the materials we used 
in studying about careers was: 



A. just about right for people of 
my age . 


.63 


.70 


B. really better for people 
younger than I am. 


.13 


.13 


C. really better for people 
older than I am. 


.25 


.17 


In making the best use of the 
materials which we used in study- 
ing careers, I thought that my 
teacher: 

A. did an especially good job. 


.65 


.49 


B. did a pretty good job. 


.21 


.34 


C. did a fair job. 


.08 


.11 


D. did not do a very good job. 


.03 


.03 


E. did a poor job. 


.03 


.03 


If we used more study materials such 
as games, posters, and films, I 
think that: 

A. I would learn alot more about 
subjects such as math, English, 
and social studies. 


.55 


.47 


B. I would learn a little more 
• about subjects such as math, 
English, and social studies. 


.24 


.26 


C. I would probably not learn 
any more about subjects such 
as math, English and social 
studies . 


.12 


.13 


D. I would probably not leani 
as much as I usually do in 
subjects such as math, English, 
and social studies. 


.09 


.14 



Table 9 , cont. 
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Questions 



Teachers 

Trained Untrained 



133. I thought that the best film was 
the one on : 



( 



135 



A. 


mass communication. 


.15 


.12 


B. 


health care. 


.32 


.39 


C. 


aerospace . 


.15 


. 17 


D. 


criminal justice. 


.30 


.25 


E. 


computer science. 


.08 


.07 


I thought that the best game 
was the one on: 






A. 


computer science. 


.06 


.07 


B. 


health care. 


.09 


.09 


C. 


mass communications. 


.08 


. 10 


D. 


criminal justice. 


.57 


.52 


E. 


aerospace. 


.20 


.22 


I thought that the best poster was 
the one on: 






A. 


computer science. 


.20 


.18 


B. 


health care. 


.17 


.17 


C. 


■mass communications. 


.22 


.24 


D. 


criminal justice. 


.26 


.24 


E. 


aerospace . 


.15 


.18 
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Table 9, cont. 
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Questions 



Teachers 

Trained Untrained 



136. 



( 



137. 



138 



In finding out about new occ- 
upations and the opportunities 
for training in two-year 
community colleges, I think 



that 


I learned the most from: 






A. 


the films. 


.48 


.53 


B. 


the games. 


.24 


.20 


C. 


the posters. 


.03 


.06 


D. 


the class discussions.-' 


.15 


.13 


E. 


what my teacher said. 


.10 


.08 


In studying about new occ- 
upations and careers, I felt that 
the most interesting materials 
were: 






A. 


the films. 


.41 


.48 


B. 


the games. 


.36 


.31 


C. 


the posters.. 


.08 


.05 


D. 


things done in class 
discussions . 


.15 


.16 


In studying about new occ- 
upations and careers, I felt 
that the most boring materials 
were: 






A. 


the fims. 


.10 


.11 


B. 


the games . 


.12 


.17 


C. 


the posters . 


.42 


.43 


D. 


things done in class 
discussions . 


.35 
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Table 9, cont. 
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139. 



( 



140. 

r 



141. 




Questions Teachers 

Trained Untrained 



In comparing how we studied new 
careers with the way we usually 
study other subjects such as 
English, math, or social studies, 
I thought that the films, games, 
and posters were: 



A. 


more boring than most 


.06 


.06 


B. 


things we use in school, 
about as interesting as 
the usual things we use 


C. 


in school. 

a little more interesting 
than the usual things we 


.15 


. 13 




use in school. 


.18 


.21 


D. 


much more interesting than 
the materials we usually use 




.23 


E. 


in school. 

very much more interesting 
than the materials we 


.18 


usually use in school. 

I thought that I learned the most 
about careers in: 


.44 


. 38 


A. 


health care. 


.25 ' 


.29 


B. 


criminal justice. 


.40 


.30 


C. 


computer science. 


.09 


.14 


D. 


mass communications . 


.11 


.10 


E. aerospace. 

I thought that I learned the 
least about careers in: 


.16 


.17 


A. 


health care. 


.19 


.15 


B. 


criminal justice. 


.11 


.15 


C. 


computer science. 


.32 


.30 


D. 


mass communications. 


.17 


.20 


E. 


aerospace. 


.20 


.21 
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Teachers' Evaluations: What were teachers evaluations of the 



materials and of in-servioe training? 

Table 10 presents a summary of teacher responses to the 10 
questions presented at the conclusion of in-service training. 

In all, responses were markedly favorable toward the training 
and the materials, and were also somewhat consistent across 
the three cities. Generalizations included: 

1. Most teacher trainees thought that the career oppor- 
tunities described in the materials were "important" for their 

t 

students; responses were less favorable in ( Houston as compared 
with Port Arthur and Conroe. , 

2. Nearly all teachers considered that the training 
provided by these materials would be "successful." 

3. On the average, over two-thirds of the teachers con- 
sidered their in-service training as "useful." 

4. Nearly all teachers considered the films as "effective. 

5. Ratings of the posters ranged mainly between being 
"average" and "very effective." 

6. Games were rated as "effective." 

7. Over half of the teachers answered a "positive yes" 
when asked if multi-media materials might be tried in other 
subject matter areas. 

8. Nearly all teachers thought that the training package 
would "work well" with their students. 

9. The "least effective" films were judged as computer 
science by Houston teachers and mass communications by Port 
Arthur teachers. (Conroe teachers did not rate the films.) 
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10. Health care was rated as the "best film" by Houston 
teachers, with mass communications a close second; aerospace 
was judged the "best film? by Port Arthur teachers. 



•i 



i 



( 



{ 



ro 



tTl 


o 


n 


w 


• 


• 


• 


• 


X 


w 


p 


Hi 


C 


o 


cr 


P 


H* 


3 


o 


H* 


rt 


ft 


c 


H 


ft 


< 


rt 


M 




P - 




X 


C 


03 


P 




P 


rt 


< 


W 


w 




ft 


C 


C 


C 


H 


o 


o 


P 


P 


o 


o 


w 


0Q 


ft 


ft 


C 


ft 


w 


w 


o 


• 


in 


w 


o 




H) 


Hi 


ft 




C 


c 


w 




h-» 


h- » 


w 




• 


* 


H> 







C 



rt h 

H H P 
O ft 
cr H-rt 

P P ft 

> c r h* h 

• m 3 9 

XOQ W 



< 


o*w o 


ft 


ft ft Hi 


H 


•• -o 


X 


C H- 
ft rt 


(A 


p (A 


C 


O 


o 


ft w 


o 


rt 


ft 


Hi P 


(a 


O rt 


w 


H ft 


Hi 


P- 



c o 

M 03 (D 

. H H* 

<* 3 
fD in 
H « 



O HH 

*13 rt 
O P - 
H H- 
rt P 
C Pc - 
P 



H* ft 
rt P - 
H* P 
ft rt 



rt 



*: P - 



w 



• • • • • 

O O M cn Isi 

© © ID O in 



o 

o 



I • * • 

© Is* © .* 

© tsi Ol .p* 



© © O © O' 

© © © cn 



1 










M 


o 


I 










• 


C 




| 












ft 


I 












. . w 


I 










H* M 


rt 


1 










P 


H* 












rt 


o 


I 










rt p- 


p 


I 










tr h* 


w 


m 


© 


n 


W 


> 


ft p 






i • 


• 


• 


• 


• 


Pf 
















3 






i 










C rt 








© 


o W 3 


3 


H* 


M p- 






c 


H 


Hi rt p 


P 


3 


rt P 






H* 


O 


C X 


X 


XI 


H* rt 






rt 


cr 


rt pa* 


cr 


o 


i 






ft 


p 


p- ft ft 


ft 


H 








| 


or 


ft P 




rt 


ft p* 






H* 


M 


9 ft h* 


H* 


P 


p ft 






H 


X 


* w B 


3 


P 


CL 






H 




* © 


XI 


rt 


h* n 






ft 


c 


rt O 


o 




p p 








5 


o cr H 


H 


H> 


H 






ft 


H* 


O C rt 


rt 


O 


3 ft 






< 


3 


• rt P 


P 


H 


p ft 






P 


xi 


P 


P 




rt H 






P 


o 


C rt 


rt 


3 


ft 






rt 


H 


P 




X 


H O 








rt 


H‘ Hi 


Hi 




H**d 






rt 


P 


3 O 


O 


w 


p *d 






° 


P 


© H 


H 


rt 


M O 








H 


O 




c 


.W H 






3 




H (A 


3 


CL 


rt 






X 


Hi 


ft o 


X 


ft 










O 


£ 9 




p 


H 5 






U> 


H 


p ft 


(A 


rt 


ft H* 






rt 




rt 


rt 


iA 


•• rt 






C 


3 


O 


C 


• 


H* 




* 


CL 


X 


rt Hi 


CL 




ft 






ft 




O 


ft 




U) 






P 


t/> 


3 


P 










rt 


ft 


w X 


rt 










(A 


c 


o 


iA 




H 






I • 


CL 


3 


• 




ft 






I 


ft 


ft 






iA 






I 


P 








ft 






I 


rt 








P 








t/i 








ft 






I 


• 








ft 
















CL 


















X 
















Zo 




I • 


• 


• 


• 


• 




1 c 




© 


O 


cn 


© 


Ol 




M in 




O 


© 


o 


© 


00 




© rt 
















O 

P 
































O 


pa 














H 


ft 


I 












rt 


w 


1 * 


• 


• 


• 


• 




Z 


X3 


© 


© 


M 


M 


•"4 




» > 


o 


o 


© 


M 


M 


00 




<£> H 


p 

(/) 






• 










ft 


1 












c 


w 














H 
















n 


1 












o 




1 • 


• 


• 


• 


• 




ZP 




© 


© 


© 


M 


00 




H H 




© 


© 


© 




.04 




© O 
















ft 






It 



... 42 



Table io: Teachers' Evaluations 
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Research Methods 
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Overview 

The basic research design can be summarized as follows: 

Within each school to be sampled comparable pairs of classes 
were selected from among fifth, seventh, and ninth grades. One 
class (experimental group) in each pair received Occupational 
Awareness Training, the second (control group) did not. All 
groups were pretested with an instrument designed to assess 
occupational knowledge and attitude variables. After the 
experimental groups received their specialized training, all 
groups were post-tested. The experimental group received an 
additional post -test questionnaire about the instructional 
materials. In each school several additional experimental 
classes were included in the study, but in these the teachers did 
not undergo training in use of the materials. 

Results were interpreted in terms of statistical comparisons 
of pre-to-post-test gains in the knowledge and attitude scores 
between experimental and control groups, and between groups 
whose teachers had or had not received training. Such compari- 
sons were made by the use of analysis of variance and appropriate 
follow-up tests, interpreting statistical significance at the 
£<.05 level. In experimental and control group comparisons, the 
analyses also included the variable of grade level thus allowing 
for the testing of interactions between effects of the materials 
and grade level. Relations of pupil sex, ethnicity, parent 
educational level, and teacher’s, attitudes toward the materials 
with gain scores were assessed'by use of special analyses. 
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Test Materials 

Development 

Main testing goals . As dictated by the aims of the field 
evaluation of the Career Awareness materials, the test instrument 
was designed to obtain primary data on five effects of the instruc 
tional package. These corresponded to: 

1. knowledge about different careers 

2. knowledge of where to get training for these careers 

3. favorability of attitude toward the importance of these 
careers 

4. favorability of attitude about the careers themselves 

5. favorability of attitude of seeing oneself in one of 
the careers 

Main test items . Test items for the five aspects of assess- 
ment were developed from an inventory of careers and attitudes 
promoted by the materials in the Career Awareness package and 
from working with small groups of children (fifth through ninth 
grades) who discussed their interpretations of the materials. 

Test items were written in a multiple choice format, and all 
were tried out and sometimes modified, based upon use with 
small groups of children in developmental activities. 

Validity and reliability . Validity of items was interpreted 
relative to the appearance of relevant information in the instruc 

t 

tional package and observation in the developmental phase that 
children (some, at least) would perceive this information. 
Reliability was interpreted in terms of children's consistency 
in answering items which were systematically repeated in develop- 
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mental activities. Eventually, reliability estimates were based 
upon correlations between pre- and post-test performances of pupils 
in the control groups in the overall data. Based upon 729 pairs 

of scores, the test-retest correlations for the five subtests were: 

» 

1. .78: knowledge about different careers 

2. .57: knowledge of where to get training for these careers 

3. .64: favorability of attitude toward the importance of 

these careers - 

4. .69: favorability of attitudes about the careers themselves 

5. .64: favorability of attitudes of seeing oneself in 

one of the careers 

Taken togethe?.* these coefficients were the basis for assuming 
that the subtest materials were acceptably reliable. 

Additional items . In order to assess pupils' subjective 
reactions to the instructional materials, selected questions were 
added to the test instrument to elicit attitudes about (1) the 
overall package, (2) films, games, and posters best liked, (3) 
the career areas that they felt they learned the most and least 
about, (4) the instructional job done by their teacher, and (5) 
whether they thought media materials of the present type would 
help them in other subjects. 

Test Format 

Demo-graphic i terns . Beyond obtaining the pupils' names, 
a 6 e » grade , sex, and classroom designation, four questions elicited 
information on: (1) father's education, (2) mother's education, 

(3) ethnic identification, and (4) occupation of the head of the 



household. 
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Main test items . Altogether there were 60 questions, 12 
for each of the five occupational areas of criminal justice, 
health care, aerospace, mass communication, and computer science. 
Again, these were in a multiple choice format. Each of these 12 
items represented a particular type of question for which the 
content was varied for each occupational area. These types were 
as follows: 

A. Occupational knowledge: 

1. definition of the area 

2. definition of a technical occupation in the area 

3. the type of person needed for a particular job 

4. what a person in a particular job does 

B. Knowledge of sources of training: 

1. training not in a community college curriculum 

2. a type of training available in a community college 

3. a further type of training available in a community 
• college , . 

C. Attitude about job importance and team importance: 

1. ratings of the importance of a job 

2. ratings of the importance of a job to a team 

D. Attitude about job favorability : 

1. rating of the interestingness of a job 

2. rating of the pride that people have in a 
particular job 

E. Attitude of self in job: 

1. ratings of the appeal of a particular job 
for oneself 
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Items for rating the materials . Twelve questions asked the 
pupils to rate various facets of the instructional materials and 
their reactions to them. These 12 questions are contained in Table 7 . 

Test booklets . All tests were typewritten and reproduced 
by photocopy process on both sides of 8Jj by 11 inch paper. 

(Copies of the tests are in the Technical Appendix to this report.) 
Brief instructions appeared at the top of the first page of 
each booklet. This was followed in all cases by questions one 
through four which were demographic items. Beyond this, there 
were three test booklet forms: 

Pre test ; all - groups : Questions 5 through 64 constituted 

the basic 60 items (12 for each occupational area) 
in a randomized sequence. These booklets were 
printed on blue stock. 

Post - test ; control - group : Questions 70 through 129 

were the same 60 basic questions as on the pre- 
test but in a different randomized order. These 
booklets were printed on light green stock. 

Pos t - test ; experimental group : Questions 70 through 129 

were the same 60 basic questions as on the pre- 
test, and items 130 through 141 were for subjective 
ratings of the materials. These were printed on 
yellow stock. 

Answer sheets . All responses were recorded' by students on 
Optical Scanning Corporation "Standard Answer Sheet-A" (see 
Technical Appendix) . 
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Teacher Training Eva luations 

At the conclusion of teacher training sessions prior to 
the administration of the Career Awareness materials in the 
classrooms, teachers completed a ten-item questionnaire asking 
them to evaluate various aspects of the training as well as 
the materials. These items were in a multiple choice format, 
and the individual questions can be seen in Table 10, a summary 
of teacher evaluations. 



Main Testing 

At the time of pretesting, the standard form answer sheets 
and the pretest booklets (blue) were distributed in classes. 

Pupils were instructed to fill in their names, grade, sex and 
a class number designation in labeled locations on the answer 
sheets. They were further given standard instructions for 

marking the multiple-choice type form. 

Pretesting was then undertaken which involved answering 
questions 1 through 64. After testing, the booklets and forms 
were taken up. Teachers stored the answer sheets for subsequent 

distribution and use in post-testing. 

Post-testing involved completion of questions 70 through 129 
in the control group booklets (green) or questions 70 through 141 
for the experimental groups (yellow) on the same answer sheets 
as used in pretesting. Sheets and booklets were then taken up and 

eventually sent for data tabulation. 

Testing was undertaken in the cities of Houston (pretest 
October 22, 1971; post-test November 9), Port Arthur (November 8, 



Data Gathering 



ERJC 





November 23), and Conroe (November 8, November 23), Texas. 

Schools in these cities included: 

Houston : Marshall Jr. High, Sharpstown Jr. High, Lanier 

Jr. High, Miller Jr. High, Patrick Henry Jr. High, Pugh Elementary, 
Backshear Elementary, White Elementary, Sutton Elementary, and 
Scroggins Elementary. 

Port Arthur: Washington Elementary, Sims Elementary, 

Pease Elementary, Lincoln, Edison, and Austin schools. 

Conroe : Sam Houston Elementary, Ronyan Elementary, 

Washington Jr. High, Travis Jr. High, and Conroe High School. 

A summary of the numbers of students tested in each city 
is: Houston, 859; Port Arthur, 501; Conroe 350. 

Data Tabulation 
Pre - tabulation Activities 

Inspection . Prior to tabulation routines, answer sheets 
were examined for such problems as missing pre- or post -test, 
improper marking of answers, obviously missing responses, obviously 
spurious responses (e.g., marking randomly or all one option), 
and extraneous marks on the sheet. When possible, a sheet was 
corrected by the research team, as when marking in a missing 
grade designation, putting in darkened marks, and the like. 

The greatest problem was missing pre- or post-tests which prevented 
including a pupil in the study. 



Keypunching. All usable answer sheets were subsequently 



submitted for automatic keypunching by optical scanning process. 
This was done by the Measurement and Evaluation Center of the 
University of Texas under a subcontract with Central Texas 
College. The response data now in keypunch card format were then 
ready for tabulation. 

Tabulation 

All tabulation was automatically undertaken by use of special 
programming p "epared for the IBM 360-40 computer system at 
Central Texas College.* Tabulation involved the following steps: 

1. Response items were read from data cards and registered 
as "1" for correct ajiswers or "0" or incorrect answers on know- 
ledge items, and for a digit directly reflecting attitudinal 
ratings. 

2. Test items were summed for each of the five basic scores 
for the pretest and for the post- test. Difference scores of 
post-test minus pre-test were calculated for each of the five 
variables. These difference scores became a part of each student's 
output record . 

3. Subjective questionnaire ratings were registered as digits 
and became a part of each student's response record. 

4. Demographic data were registered as a part of each 

!* 

student's response record. i 

5. The class identification code has 'Contained two digits 



* Programming was done by Mr . Bud Goetcher in consultation 
with Dr. Jack L* Whitehead of Austin, Texas, and under the 
supervision of the research director (f.w.). 
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reflecting teachers’ evaluations of the training materials; this 
became a part of each student's record. 

, Statistical Analyses 

Details of the statistical analyses may be found in the 
Technical Appendix to this report. Interpretations of the analyses 
were incorporated directly in the reports of the findings. The 
general strategies for analyses are summarized as follows: 

Comparison of Pre - to - Post - test Gains . 

f 

The basic analysis model for testing the statistical 
significance betweeri average score gains of the control and 
experimental groups was a two-by-three analysis of variance. 

Such analyses were conducted for each of the five main variables 
on data from each of the three test cities, and once for all 
cities combined. The results of these analyses were used in the 
subsequent comparisons of control as against experimental group 
mean gain scores for each of the fifth, seventh, and ninth 
( grades within a city. The procedures for this model of analyses 

of variance may be found in: Donald J. Veldman, Fortran 

Programming for the Behavioral Sciences (New York, 1967) . 

\ 

The same type analysis of variance, but only involving a 
direct comparison of experimental groups having trained as 
against untrained teachers, was used to test score gains relative 
to teacher training. 

Relations of Gains to Other Variables 

In order to see the degree to which variables of sex, parent 
education, occupational status, and teacher attitude were related 
to pre-to-post-test gains, analyses of variance and various 
correlational procedures were 
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calculated for each of the five basic scores on these potentially 
associated variables. This was done only for the overall data 
(combining cities) . The computational approach described by 
Veldman (see earlier reference) was used. As noted in the 
description of results, selected further statistical comparisons 
were made on an ad hoc basis when gains seemed particularly 
related to one of the above variables. These analyses are 
described in such discussions. 

Cross - Tabulation of Subjective Data 

The results of the questions asking for ratings of the 
materials were subjected to an overall cross -tabulation analysis 
on a question by question basis relative to child grade, sex, 
and ethnicity. A further set of cross - tabulations were made 
for comparisons between groups having trained or untrained 
teachers. No statistical inferences were thought necessary in 
the analyses of these cross-tabulations because patterns of 
different responses were for the most part directly interpretable . 
Cross-tabulation procedures developed by Veldman (see earlier 
reference) were used. 
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